How many relevances in information retrieval?

ion, independent from the particular data base. A possible set of ACD could be: 1. Comprehensibility: how much the retrieved documents have to be easy to understand; 2. Recency: how much the retrieved documents have to be recent; 3. Quantity: how much information the user wants (number of documents and their length); 4. Language: in which language the retrieved documents have to be written; 5. Fertility: how much the retrieved documents will be useful for nding other documents (eg. number of references of the documents). And a possible set of CCD, obtained analysing the INSPEC data base, could be: DT (Document type): type of document, for instance BC (Book Chapter), BK (Book), CA (Conference Article), JA (Journal Article), DS (Dissertation), RP (Report), and so on; TR (Treatment code): document character, for instance: A (Application), B (Bibliography), G (General or Review), T (Theoretical or Mathematical), and so on; PG (Pages): number of pages of the document; MD (Meeting date), PD (Publication date) and P5 (Original Publication Date): when the document has appeared; LA (Language): language used for writing the document. One could also take into account the abstract, that contains the number of references of the document and some other hint on how much the document is introductory, theoretical, and so on. The use of a commercial and widely available data base, as INSPEC is, and the generality of the chosen CCD (that can easily be found in other data bases) assure an immediate practical usefulness of this approach. Let us consider two examples: 1. In Italy, a university professor has to prepare in a few hours an introductory lesson on the UNIX system for a rst year course. The professor has a computer science background, but he does not remember a lot about that topic. Moreover, he needs to give a good bibliographic reference to his students. Thus, he will need a few introductory and (preferably) written in Italian documents, and he will not be interested in their recency and fertility. 2. An Italian PhD student has to prepare his PhD thesis on a topic that he does not know very well, and he wants to verify some ideas that he deems promising and original. In this case, comprehensibility is not a crucial aspect, while recency, quantity, and fertility are important. The language of the documents must be Italian or English, the only ones that he knows. 14 Table 1 summarises the ACD of the retrieved documents. `+' means that the corresponding ACD is important, `{' that it is not important, and `=' that has a medium importance. Note that these ACD derive only from the task and context components of the user's need. Professor PhD student 1. Comprehensibility High (+) High ({) 2. Recency ({) Recent (+) 3. Quantity Low (+) High (+) 4. Language Italian (=) Italian or English (+) 5. Fertility ({) High (+) Table 1: ACD of the documents in the two examples. Now let us see how to map the ACD in CCD. In the rst case (professor), we can surely reject documents with `Document type' CA or DS; the number of pages (PG) must be low, the dates (MD, PD, P5) and the number of bibliographic references are not important and the language (LA) is preferably Italian. In the second case (PhD student), the dates must be recent, the language Italian or English, and the other characteristics are not important. It seems so feasible to obtain the CCD from the ACD. The obtained CCD could be used in two ways, either for modifying the query (the professor can reject with a high certainty CA and DS documents), or for ranking the retrieved documents (old documents could anyway be interesting for the PhD student). To have a quantitative idea of the performance improvement that can be obtained by means of an IR system that models the task of the user, let us think of a user interested in documents of a certain kind (for instance, books) and let us suppose that the data base is equally partitioned into three di erent kinds of documents, for instance books, journal articles, and proceedings articles. If the task component is not taken into account, then probably at least 2/3 of the retrieved documents will be not relevant (though topical); with an IR system capable of modelling the task, the performance (actually, precision) could ideally be three times higher. 4.2.2 Presentation of information The classi cation could be useful also for the presentation of information issue. It could be possible to design a user interface to an IR system capable of visualising in some way (eg. using virtual reality techniques) the four dimensional relevance space, where the documents could be represented as points (analogously to the `star eld displays', Ahlberg and Shneiderman, 1994) and directly manipulated by the user. In such an interface, the system could present to the user the most relevant (with respect to each of the relevances) documents, and the user could browse them, read their content, move them in the relevance space, and so on. Such a direct manipulation interface, besides providing the user with an intuitive visualisation mechanism that seems to have good performances, could allow the system to have some feedback from the user. For example, the user could move (dragging with the mouse) a document from the rel(fTopic,Taskg) zone to the rel(fTopicg) zone, thus allowing the system to 15 know that some features of that document make it not suited for the task athand.4.2.3 Relevance feedbackThe classi cation takes into account the relevance feedback activity: the rele-vance expressed by the user during the interaction with the IR system is di er-ent from rel(Information;RIN; t(f); fTopic,Task,Contextg), hence a documentjudged relevant before could be not relevant later.Besides that, the classi cation could be the basis of a relevance feedbackactivity richer than the classical one. In classical relevance feedback (Harman,1992), the user judges the retrieved documents as either relevant or not relevant.But saying that a document is relevant (not relevant) with respect to a particularrelevance gives additional information that can be fruitfully used by the IR sys-tem.4 For instance, a document already known by the user is not relevant withrespect to rel(fContextg), but it is relevant with respect to rel(fTopic,Taskg).And such a document is obviously a good candidate for a positive feedback, asit contains both topical terms and characteristics suited for the task at hand.4.3 Evaluation of IR systemsWhich is the relevance to use in the IR evaluation? The classi cations of rele-vances and relevance judgements could be used as a useful framework on whichbasis to better understand the evaluation issues. Brie y, in the classical IR sys-tem evaluations (Cran eld, see Cleverdon et al., 1966, and TREC, see Harman,1993), the relevance used, rel(Surrogate;Request; fTopicg); is a \lower" one inthe ordering of Figure 6, but the attempts to climb the ordering (Borlund andIngwersen, 1997) face many problems. There seems to be a sort of \Relevanceindetermination phenomenon" (borrowing the term from quantum physics): themore we try to measure the \real" (user) relevance, the less we can measure it.So the right compromise must be found. Finally, it should be observed thatrecall and precision are still meaningful and useful aggregates for each kind ofrelevance.5 Conclusions and future workIn (Saracevic, 1996) a system of various interplaying relevances is proposed.The classi cation of various relevances presented in this paper has a narrowerrange, but it has also some important features:It is very schematic and formalised;It is useful for avoiding ambiguities on which relevance (and relevancejudgement) we are talking about;It shows how it is short-sighted to speak merely of `system relevance'(the relevance as seen by an IR system) as opposed to `user relevance'(the relevance in which the user is interested), and how `topicality' (a4This information could be communicated to the IR system by a user using a direct ma-nipulation interface similar to the one described in Section 4.2.2.16 relevance for what concerns the topic component) is conceptually di erentfrom `system relevance';It has to be considered in the implementation of IR systems working closerto the user, as it is possible to improve a classical IR system along the fourindependent directions, as above discussed. Moreover, some postulates ofimpotence (borrowing the term from Swanson, 1988) can be straightfor-wardly derived from the classi cation, for instance: rel(Request; t(qn)) isthe maximum relevance that can be handled with certainty by an IR sys-tem; rel(Surrogate) is the maximum relevance that can be handled whenusing bibliographic databases, and so on;It emphasises that it is not so strange that di erent studies on relevanceobtain di erent results (see Mizzaro, 1997b), as one should pay attentionto both which kind of relevance is measured and which kind of relevancejudgement is adopted: often this issues are not taken into account.This work is at an initial stage, and there is still a lot to be done. Possibleresearch questions could be: Are the four dimensions correct? Do we need otherdimensions? Is it possible to nd an intuitive graphical representation betterthan the one in Figure 6? Is the decomposition in topic, task, and contextcorrect? Is it possible to extend it, re ne it, and de ne it in a more formal way?Is it possible to improve in some way the orderings on the four dimensions?5Finally, in this paper an ordering has been de ned, but it might be interest-ing to nd also a metric in order to measure the distances among the variousrelevances. For doing this, I think it is mandatory to proceed in an experimentalway, confronting two or more di erent kinds of relevance. The four researchesbrie y summarised at the end of Section 4.1 (Cooper, 1973a; Cooper, 1973b;Regazzi, 1988; Saracevic et al., 1988; Saracevic and Kantor, 1988a; Saracevicand Kantor, 1988b; Brajnik et al., 1996; Mizzaro, 1997a) are some preliminarysteps in this direction. This line of research has important practical conse-quences: with such a metric, it would be possible to choose in an objective wayamong the possible directions (described in the Section 4.2.1) for developingmore e ective IR systems.AcknowledgementsThanks to Mark Magennis, Giorgio Brajnik, Marion Crehange, Peter Ingw-ersen, Giuseppe O. Longo, Carlo Tasso, and three anonymous referees for usefuldiscussions and comments on earlier versions of this paper.ReferencesAhlberg, C. and Shneiderman, B. (1994). Visual information seeking: Tightcoupling of dynamic query lters with star eld displays, ACM CHI '94Conference Proceedings, Boston, MA, pp. 313{317.5For instance, an alternative could be to use the set cardinality for ordering Comp,thus obtaining a more strict ordering (fTopicg < fTask,Contextg, and thus rel(fTopicg)rel(fTask,Contextg), while these two relevances are not comparable with the order proposedin Section 2.5).17 Barry, C. L. (1994). User-de ned relevance criteria: An exploratory study,Journal of the American Society for Information Science 45(3): 149{159.Belkin, N. J., Oddy, R. N. and Brooks, H. M. (1982a). ASK for informa-tion retrieval: Part I. Background and theory, Journal of Documentation38(2): 61{71.Belkin, N. J., Oddy, R. N. and Brooks, H. M. (1982b). ASK for informationretrieval: Part II. Results of a design study, Journal of Documentation38(3): 145{164.Borlund, P. and Ingwersen, P. (1997). The development of a method for theevaluation of interactive information retrieval systems, Journal of Docu-mentation 53(3): 225{250.Brajnik, G., Mizzaro, S. and Tasso, C. (1996). Evaluating user interfaces toinformation retrieval systems: A case study on user support, SIGIR96,19th International Conference on Research and Development in Informa-tion Retrieval, Zurich, Switzerland, pp. 128{136.Bruce, H. W. (1994). A cognitive view of the situational dynamism of user-centered relevance estimation, Journal of the American Society for Infor-mation Science 45(3): 142{148.Cleverdon, C. W., Mills, J. and Keen, M. (1966). Factors Determining thePerformance of Indexing Systems, Vol. 1: Design, Vol. 2: Test results,College of Aeronautics, Cran eld, UK.Cooper, W. S. (1971). A de nition of relevance for information retrieval, Infor-mation Storage and Retrieval 7(1): 19{37.Cooper, W. S. (1973a). On selecting a measure of retrieval e ectiveness, part 1:The \subjective" philosophy of evaluation, Journal of the American Societyfor Information Science 24(2): 87{100.Cooper, W. S. (1973b). On selecting a measure of retrieval e ectiveness, part2: Implementation of the philosophy, Journal of the American Society forInformation Science 24(6): 413{424.Cuadra, C. A. and Katter, R. V. (1967a). Experimental studies of relevancejudgements, NSF report TM-3520/001, 002, 003, Systems DevelopmentCorporation, Santa Monica, CA. 3 vols.Cuadra, C. A. and Katter, R. V. (1967b). Opening the black box of \relevance",Journal of Documentation 23(4): 291{303.Cuadra, C. A. and Katter, R. V. (1967c). The relevance of relevance assessment,Proceedings of the American Documentation Institute, Vol. 4, AmericanDocumentation Institute, Washington, DC, pp. 95{99.Foskett, D. J. (1970). Classi cation and indexing in the social sciences, ASLIBProceedings, Vol. 22, pp. 90{100.Foskett, D. J. (1972). A note on the concept of \relevance", Information Storageand Retrieval 8(2): 77{78.18 Froehlich, T. J. (1994). Relevance reconsidered|Towards an agenda for the 21stcentury: Introduction to special topic issue on relevance research, Journalof the American Society for Information Science 45(3): 124{133.Furnas, G. W., Landauer, T. K., Gomez, L. M. and Dumais, S. T. (1987). Thevocabulary problem in human-system communications, Communicationsof the Association for Computing Machinery 30(11): 964{971.Harman, D. (1992). Relevance feedback revisited, in N. Belkin, P. Ingwersen andA. M. Pejtersen (eds), Proceedings of the 15th ACM SIGIR, ACM Press,New York, pp. 1{10.Harman, D. (1993). Overview of the rst TREC conference, in R. Korfhage,E. Rasmussen and P. Willet (eds), Proceedings of the 16th ACM SIGIR,ACM Press, New York, pp. 36{47.Ingwersen, P. (1992). Information Retrieval Interaction, Taylor Graham, Lon-don.JAS (1994). Special topic issue: Relevance research, Journal of the AmericanSociety for Information Science 45(3).Lancaster, F. W. (1979). Information Retrieval Systems: Characteristics, Test-ing and Evaluation, 2nd edn, John Wiley and Sons, New York.Mackay, D. M. (1960). What makes the question, The Listener 63(5): 789{790.Mizzaro, S. (1995). Le di erenti relevance in information retrieval: una classi -cazione, Proceedings of the annual conference AICA'95, Vol. I, pp. 361{368.In Italian. Translation of the title: \The various relevances in informationretrieval: a classi cation".Mizzaro, S. (1996a). A cognitive analisys of information retrieval, in P. Ingw-ersen and N. O. Pors (eds), Information Science: Integration in Perspective| Proceedings of CoLIS2, The Royal School of Librarianship, Copenhagen,Denmark, pp. 233{250. Paper awarded with the \CoLIS2 Young ScientistAward".Mizzaro, S. (1996b). How many kinds of relevance in IR?, in M. D. Dun-lop (ed.), Proceedings of the Second Mira Workshop, Monselice, Italy.University of Glasgow Computing Science Research Report TR-1997-2,http://www.dcs.gla.ac.uk/mira/workshops/padua_procs/.Mizzaro, S. (1996c). Howmany relevances in IR?, in C. W. Johnson andM. Dun-lop (eds), Proceedings of the Workshop `Information Retrieval and HumanComputer Interaction', GIST Technical Report GR96-2, Glasgow Univer-sity, The British Computer Society, Glasgow, UK, pp. 57{60.Mizzaro, S. (1997a). Il Reperimento delle informazioni: analisi teorica e speri-mentazione, Phd thesis in Engineering of Information, University of Trieste(Italy). In Italian. The translation of the title is \Information retrieval:theoretical analysis and experimentation".Mizzaro, S. (1997b). Relevance: The whole history, Journal of the AmericanSociety for Information Science 48(9): 810{832.19 Rees, A. M. and Schulz, D. G. (1967). A eld experimental approach to the studyof relevance assessments in relation to document searching. 2 vols., NSFContract No. C-423, Center for Documentation and Communication Re-search, School of Library Science, Case Western Reserve University, Cleve-land, Ohio.Regazzi, J. J. (1988). Performance measures for information retrieval systems|An experimental approach, Journal of the American Society for Informa-tion Science 39(4): 235{251.Salton, G. (1989). Automatic Text Processing { The transformation, Analysis,and Retrieval of Information by Computer, Addison-Wesley, Reading, MA.Saracevic, T. (1975). Relevance: A review of and a framework for the thinkingon the notion in information science, Journal of the American Society forInformation Science 26(6): 321{343.Saracevic, T. (1996). Relevance reconsidered '96, in P. Ingwersen and N. O.Pors (eds), Information Science: Integration in Perspective | Proceedingsof CoLIS2, The Royal School of Librarianship, Copenhagen, Denmark,pp. 201{218.Saracevic, T. and Kantor, P. (1988a). A study of information seeking andretrieving. II. Users, questions, and e ectiveness, Journal of the AmericanSociety for Information Science 39(3): 177{196.Saracevic, T. and Kantor, P. (1988b). A study of information seeking andretrieving. III. Searchers, searches, and overlap, Journal of the AmericanSociety for Information Science 39(3): 197{216.Saracevic, T., Kantor, P., Chamis, A. and Trivison, D. (1988). A study of in-formation seeking and retrieving. I. Background and methodology, Journalof the American Society for Information Science 39(3): 161{176.Schamber, L. (1994). Relevance and information behavior, Annual Review ofInformation Science and Technology, Vol. 29, pp. 3{48.Schamber, L., Eisenberg, M. B. and Nilan, M. S. (1990). A re-examination ofrelevance: Toward a dynamic, situational de nition, Information Process-ing & Management 26(6): 755{776.Soergel, D. (1994). Indexing and retrieval performance: The logical evidence,Journal of the American Society for Information Science 45(8): 589{599.Swanson, D. R. (1977). Information retrieval as a trial-and-error process, LibraryQuarterly 47(2): 128{148.Swanson, D. R. (1986). Subjective versus objective relevance in bibliographicretrieval systems, Library Quarterly 56: 389{398.Swanson, D. R. (1988). Historical note: Information retrieval and the futureof an illusion, Journal of the American Society for Information Science39(2): 92{98.20 Taylor, R. S. (1968). Question-negotiation and information seeking in libraries,College and Research Libraries 29: 178{194.van Rijsbergen, C. J. (1979). Information Retrieval, 2nd edn, Butterworths,London.Vickery, B. C. (1959a). The structure of information retrieval systems, Pro-ceedings of the International Conference on Scienti c Information, Vol. 2,National Academy of Sciences, Washington, DC, pp. 1275{1290.Vickery, B. C. (1959b). Subject analysis for information retrieval, Proceedingsof the International Conference on Scienti c Information, Vol. 2, NationalAcademy of Sciences, Washington, DC, pp. 855{865.Wilson, P. (1973). Situational relevance, Information Storage and Retrieval9(8): 457{471.21